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ABSTRACT 

The  U.S.  Environmental  Protection  Agency  (EPA)  Refer¬ 
ence  Method  9  (Method  9)  is  the  preferred  enforcement 
approach  for  verifying  facility  compliance  with  federal 
visible  opacity  standards.  Supporters  of  Method  9  have 
cited  its  flexibility  and  low  cost  as  important  technologi¬ 
cal  and  economic  advantages  of  the  methodology.  The 
Digital  Opacity  Compliance  System  (DOCS),  an  innova¬ 
tive  technology  that  employs  digital  imaging  technology 
for  quantifying  visible  opacity,  has  been  proposed  as  a 
technically  defensible  and  economically  competitive  al¬ 
ternative  to  Method  9.  Results  from  the  field  application 
of  the  DOCS  at  EPA-approved  Method  9  smoke  schools 
located  in  Ogden,  UT,  Augusta,  GA,  and  Columbus,  OH, 


IMPLICATIONS 

Since  its  promulgation,  Method  9  has  been  the  methodol¬ 
ogy  adopted  by  both  regulators  and  permitted  facilities  to 
validate  compliance  with  federal  visible  opacity  standards. 
Because  it  depends  on  the  visual  precision  of  human  ob¬ 
servers,  Method  9  results  are  vulnerable  to  claims  of  bias, 
distortion,  and  outright  fraud.  The  DOCS,  which  generates 
a  permanent  digital  record  of  visual  emissions,  has  been 
proposed  as  an  alternative  to  Method  9.  When  sky  is  em¬ 
ployed  as  background,  the  DOCS  not  only  exhibits  com¬ 
parable  performance  flexibility  to  Method  9  but  is  also 
economically  competitive  and  can  quantify  plume  opacity 
with  an  accuracy  that  routinely  exceeds  the  level  estab¬ 
lished  by  Method  9. 


demonstrated  that,  under  clear  sky  conditions,  the  DOCS 
consistently  met  the  opacity  error  rate  established  under 
Method  9. 

Application  of  hypothesis  testing  on  the  smoke 
school  data  set  confirmed  that  the  DOCS  was  equivalent 
to  Method  9  under  clear  sky  conditions.  Under  overcast 
sky  conditions,  human  observers  seemed  to  be  more  ac¬ 
curate  than  the  DOCS  in  measuring  opacity.  However, 
within  the  smoke  school  environment,  human  observers 
routinely  employ  backgrounds  other  than  sky  (e.g.,  trees, 
telephone  poles,  billboards)  to  quantify  opacity  on  over¬ 
cast  days.  Under  conditions  that  compel  the  use  of  sky  as 
plume  background  (e.g.,  emission  stacks  having  heights 
above  the  tree  line),  the  DOCS  appears  to  be  a  more 
accurate  methodology  for  quantifying  opacity  than  are 
human  observers. 

INTRODUCTION 

The  presence  of  visible  air  emissions  from  industrial  op¬ 
erations  provides  irrefutable  evidence  that  airborne  parti¬ 
cles  are  being  discharged  into  the  atmosphere.  Not  only 
has  the  public  expressed  its  concern  regarding  the  nega¬ 
tive  psychological  effects  of  visible  emissions,  but,  under 
many  circumstances,  particulate  emissions  have  been 
identified  as  the  cause  of  increased  human  health  and 
environmental  risk.1'2  Because  of  public  concerns  regard¬ 
ing  the  potential  health  and  environmental  impacts  as¬ 
sociated  with  visible  air  emissions,  current  federal  statutes 
as  well  as  many  state  and  local  air  quality  control  laws 


296  Journal  of  the  Air  &  Waste  Management  Association 


Volume  54  March  2004 


Report  Documentation  Page 


Form  Approved 
OMB  No.  0704-0188 


Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 


1.  REPORT  DATE 

MAR  2004 


2.  REPORT  TYPE 


3.  DATES  COVERED 

00-00-2004  to  00-00-2004 


4.  TITLE  AND  SUBTITLE 

Measuring  Visual  Opacity  Using  Digital  Imaging  Technology 


5a.  CONTRACT  NUMBER 


5b.  GRANT  NUMBER 


5c.  PROGRAM  ELEMENT  NUMBER 


6.  AUTHOR(S) 


5d.  PROJECT  NUMBER 


5e.  TASK  NUMBER 


5f.  WORK  UNIT  NUMBER 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS (ES) 

Engineering  and  Services  Research  Laboratory, Tyndall  Air  Force 
Base, Panama  City ,FL, 32403 


8.  PERFORMING  ORGANIZATION 
REPORT  NUMBER 


9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS  (ES) 


10.  SPONSOR/MONITOR’S  ACRONYM(S) 


11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 


12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 


13.  SUPPLEMENTARY  NOTES 

J.  Air  &  Waste  Manage.  Assoc.  54:296-306,  March  2004 


14.  ABSTRACT 


15.  SUBJECT  TERMS 


17.  LIMITATION  OF 

18.  NUMBER 

ABSTRACT 

OF  PAGES 

Same  as 

14 

Report  (SAR) 

16.  SECURITY  CLASSIFICATION  OF: 


a.  REPORT 

unclassified 


b.  ABSTRACT 

unclassified 


c.  THIS  PAGE 

unclassified 


19a.  NAME  OF 
RESPONSIBLE  PERSON 


Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


McFarland  et  at. 


currently  regulate  the  opacity  of  plumes  (i.e.,  point 
sources  of  air  pollution)  as  well  as  fugitive  particulate 
emissions.  Opacity  is  defined  as  the  percent  of  light  at¬ 
tenuated  by  airborne  emissions.  In  other  words,  an  opac¬ 
ity  value  of  10%  indicates  that  10%  of  the  incident  light 
is  absorbed,  scattered,  or  otherwise  blocked  by  the  plume, 
while  90%  is  transmitted.  The  federal  opacity  standards 
for  emissions  from  various  industries  are  found  in  40  CFR 
Part  60  (Standards  of  Performance  for  New  and  Modified 
Stationary  Sources)  and  40  CFR  Parts  61  and  62  (Emission 
Standards  for  Hazardous  Air  Pollutants).2 

The  primary  method  for  determining  compliance 
with  federal  opacity  standards  is  U.S.  Environmental  Pro¬ 
tection  Agency  (EPA)  Reference  Method  9  (i.e.,  Method 
9 — Visual  Determination  of  the  Opacity  of  Emissions  for 
Stationary  Sources).  Method  9  relies  on  the  ability  of 
trained  human  observers  to  visually  estimate  the  opacity 
of  a  plume  by  taking  a  series  of  opacity  measurements  at 
the  rate  of  one  every  15  sec  for  a  specified  period  of  time 
(typically  6-60  min).3'4  The  collective  set  of  opacity  mea¬ 
surements  then  is  averaged  to  develop  a  single  opacity 
reading  for  the  regulated  source,  which  is  subsequently 
compared  against  permitted  levels.  To  qualify  as  an  EPA 
Reference  Method  9-certified  human  observer,  an  individ¬ 
ual  must  attend  and  successfully  complete  an  EPA- 
approved  Reference  Method  9  visual  opacity  smoke 
school  once  every  6  months.  EPA  certification  requires 
that  each  candidate  complete  EPA  Reference  Method  9 
classroom  and  field  training  and  receive  a  passing  grade 
on  an  opacity  field  test.  To  successfully  pass  the  opacity 
field  test,  the  candidate  must  demonstrate  the  ability  to 
accurately  assign  an  opacity  value  to  each  of  25  white  and 
25  black  smoke  plumes  with  an  error  rate  not  to  exceed 
7. 5%. 5 

While  Method  9  has  an  extensive  history  of  success¬ 
ful  employment,  the  costs  associated  with  smoke  school 
tuition  fees,  an  attendee's  travel  and  lodging,  and  the  3 
days  that  employees  are  typically  absent  from  their  pri¬ 
mary  work  assignment  support  the  claim  that  maintain¬ 
ing  active  certification  can  be  expensive.  Beyond  the  fi¬ 
nancial  burden  associated  with  an  individual  maintaining 
EPA  Reference  Method  9  certification,  reliance  on  human 
observers  to  estimate  visible  opacity  suggests  that  the 
results  are  inherently  susceptible  to  bias  and  other  factors 
that  could  potentially  affect  their  accuracy.  Moreover,  it  is 
not  uncommon  for  a  facility's  air  compliance  personnel 
and  government  regulators  to  arrive  at  significantly  dif¬ 
ferent  results  when  applying  Method  9  procedures  to 
estimate  the  opacity  of  a  permitted  source.6 

A  major  challenge  in  estimating  visible  opacity  is  the 
impact  of  particle  concentration,  particle  size  distri¬ 
bution,  particle  optical  properties,  solar  illumination 
angle,  and  moisture  content  of  the  plume  on  the 


measurement.4'5  To  minimize  the  impact  of  these  and 
other  variables  on  the  opacity  determination,  large  indus¬ 
trial  operations  often  install  sophisticated  optical  instru¬ 
ments  called  transmissometers  to  provide  continuous 
opacity  monitoring.  When  properly  installed  and  cali¬ 
brated,  transmissometers  provide  effective,  real-time 
opacity  measurements.  However,  the  capital  and  mainte¬ 
nance  costs  associated  with  the  installation  and  operation 
of  these  instruments  can  be  prohibitive  economically. 
Moreover,  like  all  automated  compliance-monitoring  de¬ 
vices,  these  instruments  require  that  a  permitted  source 
be  equipped  with  a  reliable  alternative  method  for  esti¬ 
mating  opacity  in  case  of  instrument  failure.  EPA  Refer¬ 
ence  Method  9  is  most  frequently  the  default  alternative 
method  for  estimating  opacity. 

The  current  study  was  designed  to  evaluate  the  tech¬ 
nical  performance  of  the  Digital  Opacity  Compliance  Sys¬ 
tem  (DOCS),  a  technology  that  has  been  proposed  as  a 
potential  cost-effective  alternative  to  EPA  Reference 
Method  9  for  measuring  opacity.7  The  DOCS  uses  a  com¬ 
mercial  off-the-shelf  digital  camera  to  capture  images  of 
visible  opacity,  which  are  then  downloaded  to  a  standard 
personal  computer  and  analyzed  using  computer  soft¬ 
ware.  The  DOCS  has  not  only  been  advertised  as  an  accu¬ 
rate  and  reliable  alternative  technology  for  quantifying 
opacity  but  has  the  added  advantage  of  furnishing  the 
user  with  a  permanent  visual  record  of  the  emissions. 

BACKGROUND 

Theory 

In  the  DOCS,  digital  photographs  of  visible  emissions  are 
taken  from  valid  positions  according  to  EPA  Reference 
Method  9  specifications.7  Once  downloaded  to  a  com¬ 
puter  on  which  the  DOCS  computer  software  has  been 
installed,  the  digital  images  can  be  evaluated  for  opacity. 
The  initial  steps  in  analyzing  the  digital  image  for  opacity 
include  (1)  activating  the  DOCS  opacity  computer  pro¬ 
gram,  (2)  retrieving  those  digital  photographs  that  are  to 
be  evaluated,  and  (3)  using  the  computer  program  to  draw 
an  analysis  box  (or  grid)  around  that  portion  of  the  visible 
emissions  that  will  be  analyzed  (Figure  1). 

After  selection  of  the  analysis  box,  the  DOCS  software 
must  distinguish  that  part  of  the  digital  image  that  corre¬ 
sponds  to  the  visible  emissions  from  that  which  corre¬ 
sponds  to  background  (i.e.,  sky  conditions).  By  assuming 
the  pixels  located  at  the  outside  of  the  analysis  box  cor¬ 
respond  to  background  and  those  located  in  the  center 
represent  visible  emissions,  the  software  is  able  to  deter¬ 
mine  whether  the  emissions  are  lighter  or  darker  than 
background. 

The  DOCS  software  basically  begins  calculation  of  the 
plume  opacity  scale  by  assigning  0%  opacity  to  those 
pixels  within  the  image  whose  maximum  saturation  value 
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Figure  1.  The  analysis  box  drawn  around  the  visible  emissions. 

is  found  to  equal  the  saturation  value  of  the  background 
color.  Through  statistical  regression,  the  software  then 
establishes  an  opacity  scale  that  represents  the  saturation 
values  of  each  pixel.  By  projecting  the  saturation  values  of 
each  pixel  onto  the  opacity  scale,  the  software  estimates 
the  opacity  of  every  pixel,  which  is  combined  to  yield 
plume  opacity. 

Practice 

Under  normal  circumstances,  the  user  of  the  DOCS  soft¬ 
ware  rarely  needs  to  understand  the  complex  mathemat¬ 
ical  relationships  associated  with  digital  image  transfor¬ 
mation.  Rather,  the  software  user  simply  draws  an 
analysis  box  around  the  area  of  the  plume  to  be  analyzed 
for  opacity.  The  DOCS  computer  software  then  selects  the 
purity  of  color  (i.e.,  saturation)  that  best  corresponds  to 
the  background  based  on  the  spectral  information  con¬ 
tained  in  the  pixels  located  at  the  edge  of  the  analysis  box. 
The  size  and  shape  of  the  analysis  box,  which  is  con¬ 
trolled  by  the  user  of  the  software,  must  be  chosen  judi¬ 
ciously  because  the  final  opacity  measurement  ultimately 
will  depend  on  what  part  of  the  image  the  DOCS  software 
identifies  as  background. 

An  important  feature  available  within  the  DOCS  soft¬ 
ware  package  is  the  brush  function,  which  essentially 
allows  the  photograph  analyzer  the  option  of  electroni¬ 
cally  selecting  any  segment  of  the  digital  image  as  back¬ 
ground.  The  analyzer  then  brushes  the  selected  back¬ 
ground  on  both  sides  of  the  plume.  The  DOCS  computer 
algorithm  then  compares  the  contrast  between  the  plume 
and  the  selected  background  to  generate  an  opacity  read¬ 
ing. 

METHODOLOGY 

In  developing  the  scope  of  the  field  study,  it  was  recog¬ 
nized  that  the  ability  of  the  DOCS  to  estimate  plume 
opacity  accurately  could  vary  significantly  depending  on 


climatic  conditions.  In  other  words,  temperature,  humid¬ 
ity,  and  other  environmental  factors  could  potentially 
affect  the  accuracy  of  the  DOCS.  To  estimate  the  extent  to 
which  environmental  conditions  influenced  the  ability  of 
the  DOCS  to  measure  accurately  plume  opacity,  the 
DOCS  was  evaluated  at  geographical  locations  represent¬ 
ing  a  range  of  climatic  conditions,  which  included  EPA- 
approved  Reference  Method  9  smoke  schools  conducted 
in  the  states  of  Utah,  Georgia,  and  Ohio. 

Field  Activities 

During  the  field  study,  four  commercially  available  digital 
cameras  (e.g.,  Kodak  DC290  or  Kodak  DC265)  were  em¬ 
ployed  to  photograph  visible  emissions  generated  as  part 
of  the  EPA  Reference  Method  9  certification  smoke 
school.  The  DOCS  photographic  imaging  software  was 
installed  and  tested  on  each  of  the  cameras  before  any 
photographs  were  taken.  Use  of  the  DOCS  camera  soft¬ 
ware  is  a  security  requirement  that  essentially  guarantees 
that  the  digital  photograph  cannot  be  altered  before  an 
opacity  determination.  No  technical  adjustments  or  phys¬ 
ical  modifications  of  the  cameras  were  necessary  to  oper¬ 
ate  the  DOCS  photographic  imaging  software. 

Each  camera  was  positioned  on  a  tripod  to  provide  a 
clear  view  of  the  visible  emissions.  The  minimum  dis¬ 
tance  of  the  cameras  from  the  stack  was  equivalent  to  at 
least  three  stack  heights  with  the  sun  oriented  in  the  140° 
sector  to  the  back  of  the  camera/observer  (Figure  2).  These 
field  procedures  were  adopted  to  be  consistent  with  the 
published  requirements  for  valid  EPA  Reference  Method  9 
visible  emissions  opacity  measurements.5 

Two  cameras  were  placed  directly  in  line  between  the 
sun  and  the  smoke  stack  while  each  of  the  two  remain¬ 
ing  cameras  was  placed  at  the  maximum  allowable  azi¬ 
muth  angle  (see  Figure  2).  In  addition  to  the  four  digital 


Figure  2.  Positioning  of  the  DOCS  cameras  (Cl ,  C2,  C3,  and  C4)  and 
visual  observers  during  the  field  study. 
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cameras  used  during  the  DOCS  evaluation,  an  EPA 
Method  9-certified  smoke  reader  (i.e.,  human  observer) 
was  assigned  to  each  camera  location  to  read  plume  opac¬ 
ities  during  the  first  day  of  the  DOCS  testing.  The  objec¬ 
tive  of  assigning  a  limited  number  of  EPA  Reference 
Method  9-certified  human  observers  to  measure  plume 
opacity  during  field  testing  was  to  establish  a  comparison 
between  the  relative  accuracy  of  certified  human  observ¬ 
ers  and  the  DOCS.  Unlike  the  EPA  Reference  Method  9 
certification  field-testing  procedures,  the  EPA  Reference 
Method  9-certified  human  observers  were  not  provided 
an  opportunity  to  calibrate  their  vision  before  estimating 
plume  opacity.  The  EPA-certified  smoke  generator  that 
was  used  during  the  subsequent  EPA  Reference  Method  9 
certification  field  test  initially  was  employed  in  the  1-day 
certified  human  observer/DOCS  side-by-side  evaluation. 
EPA  Reference  Method  9-certified  human  observers  who 
were  scheduled  to  participate  in  the  1-day  technology 
comparison  had  active  EPA  Reference  Method  9  certifica¬ 
tions  at  the  beginning  of  the  fieldwork. 

Weather  Monitoring 

During  the  EPA-approved  smoke  school  DOCS  evalua¬ 
tion,  on-site  field  personnel  recorded  the  values  of  specific 
climatic  parameters  including  (1)  mean  air  temperature, 
(2)  average  wind  speed,  (3)  maximum  wind  speed,  (4) 
wind  direction,  (5)  sky  conditions,  (6)  relative  humidity, 
(7)  visibility,  (8)  barometric  pressure,  (9)  precipitation, 
(10)  horizontal  sun  angle,  and  (11)  vertical  sun  angle 
during  each  day  of  testing.  Methods  used  to  estimate  the 
value  of  each  climatic  parameter  are  summarized  in 
Table  1. 

Table  1 .  Methods  used  to  estimate  the  value  of  various  climatic  parameters. 


Opacity  Determination  Using  the  DOCS 

After  collecting  the  digital  photographs  of  each  plume, 
opacity  was  estimated  using  the  DOCS  computer  software 
by  an  eight-member  panel  consisting  of  federal  govern¬ 
ment  civilian  personnel,  U.S.  military  personnel,  and  fed¬ 
eral  government  contractors.  Each  panel  member  was 
provided  a  compact  disc  that  contained  all  of  the  digital 
photographs  taken  from  the  respective  smoke  school  as 
well  as  the  DOCS  computer  software  and  user  guide.  The 
panel  members  were  required  to  work  independently  to 
estimate  the  plume  opacity  of  each  digital  photograph 
using  the  furnished  computer  software.  Once  panel  mem¬ 
bers  had  completed  their  analyses,  the  opacity  results 
were  transferred  and  stored  electronically  in  a  relational 
database  for  subsequent  statistical  evaluation.  An  inde¬ 
pendent  quality  control  officer  was  assigned  the  respon¬ 
sibility  of  maintaining  the  integrity  of  all  opacity  data, 
including  the  opacity  results  generated  from  the  EPA- 
approved  transmissometer  against  which  the  DOCS  and 
EPA  Reference  Method  9-certified  human  observer  opac¬ 
ity  data  were  compared. 

In  evaluating  the  DOCS  as  a  technically  defensible 
alternative  to  EPA  Reference  Method  9,  the  mean  opacity 
deviation  (i.e.,  difference  in  opacity  readings  recorded  by 
the  DOCS  and  the  EPA-certified  transmissometer)  for 
both  black  and  white  smoke  plumes  was  computed  and 
compared  with  the  EPA  Reference  Method  9  statistical 
error  rate  of  7.5%.3'5  Moreover,  to  ensure  that  technolog¬ 
ical  decisions  resulting  from  the  field  data  could  be  sup¬ 
ported  with  a  known  degree  of  confidence,  the  uncer¬ 
tainty  associated  with  the  opacity  deviation  measurement 
was  quantified  by  computing  both  the  95  and  99%  con¬ 
fidence  intervals.  Finally,  conclusions  regarding  the 
equivalency  of  the  DOCS  to  EPA  Reference  Method  9  were 
drawn  from  the  application  of  statistical  hypothesis  test¬ 
ing  on  the  smoke  school  field  data. 


Parameter 


Method 


Mean  temperature 
Average  wind  speed 
Max  wind  speed 

Wind  direction 
Sky  conditions 
Relative  humidity 
Visibility 

Barometric  pressure 

Precipitation 

Horizontal  sun  angle 
Vertical  sun  angle 


Standard  liquid  thermometer  (Eastern  Technical  Associates,  Inc.) 
Standard  anemometer  (Eastern  Technical  Associates,  Inc.) 
National  Weather  Service  (National  Oceanic  and  Atmospheric 
Administration — -www.nws.noaa.gov) 

Standard  anemometer  (Eastern  Technical  Associates,  Inc.) 

Visual  observation 

Sling  psychrometer  (Eastern  Technical  Associates,  Inc.) 

National  Weather  Service  (National  Oceanic  and  Atmospheric 
Administration — -www.nws.noaa.gov) 

National  Weather  Service  (National  Oceanic  and  Atmospheric 
Administration — -www.nws.noaa.gov) 

National  Weather  Service  (National  Oceanic  and  Atmospheric 
Administration — -www.nws.noaa.gov) 

Magnetic  compass  (Eastern  Technical  Associates,  Inc.) 

Abney  level  (Eastern  Technical  Associates,  Inc.) 


RESULTS 

The  following  section  summarizes  the  statistical  analysis 
of  the  DOCS  field  data  collected  at  each  of  the  three 
EPA-approved  Reference  Method  9  smoke  schools.  To  pro¬ 
vide  technology  decision-makers  with  a  rational  basis  for 
drawing  conclusions  regarding  the  equivalency  of  the 
DOCS  to  EPA  Reference  Method  9,  the  field  results  were 
evaluated  using  a  standard  statistical  hypothesis  testing 
approach. 


Ogden,  UT,  Smoke  School 

The  Ogden,  UT,  smoke  school  field  evaluation  was  con¬ 
ducted  over  a  3-day  period  from  October  2  through  Oc¬ 
tober  4,  2001.  The  climatic  conditions  over  the  3  days  of 
the  smoke  school  field  trial  are  reported  in  Table  2.  In 
general,  the  weather  conditions  during  the  Utah  field  tests 
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Table  2.  Climatic  parameters  measured  or  reported  during  the  DOCS  field  tests — 
Ogden. 


Day  of  Test 

Day  1 

Day  2 

Day  3 

Mean  temperature  (°F) 

66.2 

60.8 

60.8 

Average  wind  speed  (mph) 

8.9 

9.7 

8.1 

Max  wind  speed  (mph) 

13.8 

16.1 

11,4 

Wind  direction3 

N-NW 

E-SE 

S-SE 

Sky  conditions 

Clear 

Clear 

Clear 

Relative  humidity  (%) 

27.2 

45.2 

30.5 

Visibility  (mi) 

7 

7 

7 

Barometric  pressure  (in.  of  Hg) 

30.08 

30.11 

30.01 

Precipitation  (in) 

0 

0 

0 

Vertical  sun  angle  (°) 

42.2 

39.4 

38.5 

aN-NW  (north-by-northwest),  E-SE  (east- by-southeast),  S-SE  (south-by-southeast). 

were  ideal.  The  climatic  conditions  were  characterized  by 
(1)  an  average  daily  temperature  range  of  60-67  °F,  (2) 
low  humidity,  and  (3)  light  wind.  Of  greater  importance 
in  the  measurement  of  visual  opacity  was  the  fact  that  sky 
conditions  were  clear,  providing  maximum  color  contrast 
between  the  plume  and  background. 

Approximately  6400  opacity  readings  of  visible  emis¬ 
sions  were  taken  as  part  of  the  DOCS  field  evaluation  in 
Utah.  Of  that  number,  4741  opacity  readings  (2336  black 
plumes  and  2405  white  plumes)  or  nearly  75%  of  those 
computed  were  deemed  acceptable  for  the  DOCS  statisti¬ 
cal  analysis.  The  decision  to  exclude  2161  opacity  read¬ 
ings  from  the  DOCS  statistical  analyses  was  based  on  a 
number  of  technical  problems  including  (1)  physical  ob¬ 
struction  of  the  smoke  plume  (e.g.,  trees,  clouds,  tele¬ 
phone  poles),  (2)  folding,  twisting,  or  other  significant 
physical  disruptions  to  the 
plume,  and  (3)  inappropriate 
modification  of  digital  image 
through  use  of  the  brush  func¬ 
tion  that  was  available  in  the 
DOCS  software  package.7 

Although  there  were  a  myr¬ 
iad  of  reasons  for  eliminating 
certain  photographs  from  opac¬ 
ity  analysis,  the  overwhelming 
majority  of  photographs  that 
were  deemed  invalid  during  the 
Ogden  field  trial  were  elimi¬ 
nated  because  of  the  use  of  the 
DOCS  brush  function  by  the 
photograph  readers.  Because  a 
consensus  could  not  be  reached 
among  the  research  partici¬ 
pants  as  to  the  level  of  bias  that 


potentially  could  be  introduced  by  the  use  of  the  brush 
function,  a  decision  was  made  to  exclude  all  opacity  read¬ 
ings  developed  with  the  assistance  of  the  brush  function 
from  the  statistical  analyses.  As  a  result  of  the  fact  that 
this  decision  was  made  after  collection  of  the  DOCS  read¬ 
ings,  over  20%  of  the  initial  data  set  from  the  Ogden  field 
tests  had  to  be  excluded  from  statistical  analysis.  For  both 
the  Georgia  and  Ohio  smoke  schools,  the  DOCS  photograph 
readers  were  instructed  not  to  use  the  bmsh  function. 

Quantitative  Analysis.  The  three  statistical  parameters 
identified  as  critical  in  the  technical  evaluation  of  the 
DOCS  were  the  (1)  absolute  value  of  the  mean  opacity 
deviation  (e.g.,  opacity  measurement  recorded  by  DOCS 
minus  the  opacity  measurement  reported  by  the  EPA- 
certified  transmissometer),  (2)  95%  confidence  interval  of 
the  mean  opacity  deviation,  and  (3)  99%  confidence  in¬ 
terval  of  the  mean  opacity  deviation.  Table  3  summarizes 
the  value  of  these  parameters  calculated  from  the  Ogden 
DOCS  field  tests. 

It  is  important  to  note  that,  in  addition  to  comparing 
the  DOCS  opacity  measurements  to  the  EPA-certified 
transmissometer  readings,  the  mean  deviation  estimated 
from  opacity  readings  recorded  by  EPA  Reference  Method 
9-certified  human  observers  also  are  included  in  Table  3 
for  comparison.  The  large  difference  in  the  number  of 
readings  recorded  by  the  DOCS  relative  to  the  number 
reported  by  EPA  Reference  Method  9-certified  human  ob¬ 
servers  primarily  was  because  of  two  aspects  of  the  exper¬ 
imental  design,  including  the  fact  that  (1)  one  digital 
photograph  could  be  evaluated  by  as  many  as  eight  DOCS 
photograph  readers  (i.e.,  maximum  number  of  members 
on  panel)  while  the  certified  human  observer  only  pro¬ 
vided  one  opacity  reading  per  plume,  and  (2)  certified 


Table  3.  Statistical  data  summary  of  Ogden  smoke  school  DOCS  evaluation. 


Color  of  Smoke- 

Opacity 

Mean  Deviation 

Number  of 

Opacity  Measurement  Approach 

Range 

(%) 

Samples 

95%  Cla 

99%  Clb 

Black— DOCS 
Black — certified  observers 
Black— DOCS 
Black — certified  observers 
Black— DOCS 
Black — certified  observers 
-DOCS 

-certified  observers 
-DOCS 

-certified  observers 
-DOCS 

-certified  observers 


White- 

White- 

White- 

White- 

White- 

White- 


0-100% 

0-100% 

0-60% 

0-60% 

0-40% 

0-40% 

0-100% 

0-100% 

0-60% 

0-60% 

0-40% 

0-40% 


6.4 

6.7 

5.6 

5.4 

5.4 

4.8 
10 

8.5 

6.7 
8.2 

5.9 
7.4 


2336 

246 

1957 

212 

1745 

194 

2405 

282 

1897 

224 

1686 

199 


6. 1- 6. 7 

5. 7- 7. 8 

5. 3- 5. 9 

5. 5- 6. 4 
5-5.7 

3. 8- 5. 7 

9.5- 10.5 

7. 5- 9. 6 

6.3- 7 
7-9.4 

5. 5- 6.2 

6. 1- 8. 7 


6-6.8 

5.4- 8. 1 
5.2-6 

4. 1- 6. 8 
4. 9-5. 8 

3.5- 6. 1 

9.4- 10.6 

7.1- 10 

6. 2- 7. 2 

6. 6- 9. 8 

5. 4- 6. 3 

5.7- 9 


a95%  Cl — 95%  confidence  interval;  b99%  Cl — 99%  confidence  interval. 
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human  observers  only  provided  visual  readings  during 
the  first  day  of  the  3-day  field  test. 

Over  the  full  range  of  opacity  (i.e.,  0-100%),  the 
mean  deviation  recorded  by  the  DOCS  for  black  smoke 
was  estimated  to  be  6.4%  with  95  and  99%  confidence 
intervals  that  ranged  from  6.1  to  6.7%  and  6  to  6.8%, 
respectively.  These  results  demonstrated  that,  for  black 
smoke,  the  DOCS  exhibited  an  opacity  error  rate  that  was 
significantly  less  than  the  error  rate  associated  with  EPA 
Reference  Method  9  (i.e.,  7.5%).3'5  Moreover,  the  range  of 
the  99%  confidence  interval  indicates  that  there  is  less 
than  a  1%  chance  of  the  true  mean  deviation  being 
greater  than  6.8%.  These  statistical  results  support  the 
conclusion  that  the  accuracy  of  the  DOCS  to  quantify  the 
visible  opacity  of  black  smoke  is  equal  to  or  greater  than 
that  established  for  the  EPA  Reference  Method  9  proce¬ 
dure.  Similarly,  in  evaluating  the  ability  of  certified  smoke 
readers  to  measure  the  opacity  of  black  smoke  over  the 
full  opacity  range,  the  mean  deviation  (e.g.,  certified  hu¬ 
man  observer  opacity  reading  minus  the  transmissometer 
opacity  reading)  was  estimated  to  be  6.7%.  Although  the 
mean  deviation  was  below  the  7.5%  error  rate  associated 
with  the  EPA  Reference  Method  9  procedure,  the  95% 
confidence  interval  included  deviations  above  7.5%,  a 
fact  that  suggests  that  there  would  be  a  high  degree  of 
uncertainty  associated  with  the  conclusion  that  the  cer¬ 
tified  human  readers  could  reliably  maintain  the  EPA 
Reference  Method  9  accuracy  level. 

In  contrast  to  black  smoke,  the  field  data  indicated 
that,  over  the  full  range  of  opacity  (0-100%),  neither  the 
DOCS  nor  the  certified  human  observers  could  achieve 
the  EPA  Reference  Method  9  accuracy  standard  when 
measuring  the  opacity  of  white  plumes.  Both  opacity 
measurement  approaches  yielded  mean  opacity  devia¬ 
tions  that  were  significantly  greater  than  the  error  rate 
associated  with  EPA  Reference  Method  9  (i.e.,  7.5%).  The 
failure  of  the  DOCS  to  accurately  measure  white  plumes 
over  the  full  range  of  opacity  is  suspected  to  stem  from 
the  inability  of  the  DOCS  computer  software  to  exclude 
the  effects  of  shadows,  which  are  characteristic  of  high- 
opacity  white  plumes  (i.e.,  those  having  opacities  of 
greater  than  60%). 7  Current  investigations  are  being  con¬ 
ducted  to  determine  if  modifications  can  be  made  to  the 
DOCS  computer  software  to  effectively  reduce  the  impact 
of  shadows  on  visible  opacity  determinations. 

With  respect  to  human  observers,  it  was  interesting 
to  note  that  not  only  did  the  certified  readers  fail  to 
achieve  the  EPA  Reference  Method  9  opacity  error  rate 
during  the  measurement  of  white  plumes,  but  the  vari¬ 
ability  in  their  measurements  (as  reflected  in  both  the  95 
and  99%  confidence  intervals)  was  appreciably  larger 
than  that  reported  by  the  DOCS.  In  other  words,  for  the 
weather  conditions  that  prevailed  at  the  Ogden  field  site, 


the  DOCS  results  appear  to  be  more  consistent  and  repro¬ 
ducible  than  the  opacity  measurements  reported  by  EPA 
Reference  Method  9-certified  human  observers. 

Despite  the  failure  of  the  DOCS  to  meet  the  EPA 
Reference  Method  9  opacity  error  rate  requirement  during 
the  measurement  over  the  full  opacity  range  for  white 
smoke  plumes,  when  the  statistical  procedures  were  ap¬ 
plied  to  a  limited  range  of  visible  opacity,  the  DOCS  not 
only  met  the  accuracy  requirement  but  was  found  to  have 
a  significantly  greater  accuracy  than  EPA  Reference 
Method  9-certified  human  observers.  For  example,  over 
the  0-60%  opacity  range  for  white  smoke,  the  mean 
deviation  of  the  DOCS  was  estimated  to  be  6.7%  with  95 
and  99%  confidence  intervals  that  ranged  from  6.3  to  7% 
and  6.2  to  7.2%,  respectively.  These  data  demonstrate 
that,  when  measuring  the  opacity  of  white  plumes  having 
opacity  levels  equal  to  or  less  than  60%,  the  DOCS  accu¬ 
racy  was  significantly  better  than  that  of  EPA  Reference 
Method  9.  Furthermore,  the  99%  confidence  interval  in¬ 
dicates  that  there  is  less  than  a  1%  chance  that  the  true 
mean  deviation  will  be  greater  than  7.2.  These  statistical 
results  support  the  conclusion  that  the  accuracy  of  the 
DOCS  in  measuring  the  opacity  of  white  smoke  over  the 
range  of  0-60%  opacity  is  significantly  greater  than  that 
established  for  EPA  Reference  Method  9.  In  contrast  to  the 
ability  of  the  DOCS  to  meet  the  EPA  Reference  Method  9 
accuracy  requirements  when  its  application  was  limited 
to  the  0-60%  opacity  range  for  white  plumes,  results 
from  the  certified  human  observers  were  significantly 
larger  than  that  permitted  by  EPA  Reference  Method  9. 
For  example,  over  the  limited  opacity  range  of  0-60%, 
the  mean  opacity  deviation  associated  with  EPA  Reference 
Method  9-certified  smoke  readers  was  estimated  to  be 
8.2%  with  a  95%  confidence  interval  that  ranged  from  7 
to  9.4%.  Clearly,  under  the  clear  sky  conditions  encoun¬ 
tered  in  Utah,  the  DOCS  was  determined  to  be  a  more 
accurate  and  precise  opacity  measurement  method  than 
were  certified  human  observers. 

Augusta,  GA,  Smoke  School 

The  EPA-approved  Georgia  smoke  school  was  conducted 
over  3  days  from  October  30  through  November  1,  2001. 
The  climatic  conditions  recorded  during  that  time  period 
are  recorded  in  Table  4.  The  weather  conditions  during 
the  smoke  school  field  tests  were  characterized  by  mild 
temperatures,  moderate  winds,  scattered  clouds,  and 
partly  overcast  skies. 

Not  only  were  the  weather  conditions  in  Augusta, 
GA,  found  to  be  appreciably  different  from  those  encoun¬ 
tered  in  Ogden,  but  the  physical  landscape  of  the  two 
smoke  school  locations  were  drastically  different  as 
well.  For  example,  while  the  Ogden  smoke  school  field 
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Table  4.  Climatic  parameters  measured  or  reported  during  the  DOCS  field  tests — 
Augusta. 


Day  of  Test 

Day  1 

Day  2 

Day  3 

Mean  temperature  (°F) 

63.1 

60 

67.5 

Average  wind  speed  (mph) 

4 

5.3 

5 

Max  wind  speed  (mph) 

4.6 

11.5 

9.2 

Wind  direction 

N 

N 

S-SE 

Sky  conditions 

Partly  cloudy 

Scattered  clouds 

Cloudy 

Relative  humidity  (%) 

50.9 

72 

90.9 

Visibility  (mi) 

10 

10 

7.9 

Barometric  pressure  (in.  of  Hg) 

30.36 

30.42 

30.27 

Precipitation  (in) 

0 

0 

0 

Vertical  sun  angle  (°) 

32.4 

31 

28.6 

tests  were  conducted  in  a  large  open  parking  lot  located 
adjacent  to  a  university  athletic  stadium,  the  venue  for 
the  Georgia  smoke  school  was  a  much  smaller  and  se¬ 
cluded  parking  lot  surrounded  by  a  dense  pine  forest.  The 
combination  of  cloudy  conditions  and  tall  trees  in  the 
vicinity  of  the  smoke  generator  provided  a  more  variable 
background  against  which  the  DOCS  and  EPA  Reference 
Method  9-certified  human  observers  were  expected  to 
estimate  plume  opacity. 

Quantitative  Analysis.  Unlike  in  the  Utah  smoke  school 
field  test,  in  the  analysis  of  the  Georgia  smoke  school 
data,  use  of  the  DOCS  brush  function  was  not  permitted 
in  the  analysis  of  any  digital  photographs.  Furthermore, 
with  several  of  the  DOCS  camera  operators  certified  as 
EPA  Reference  Method  9  visual  observers,  a  larger  number 
of  EPA  Reference  Method  9-certified  readers  were  avail¬ 
able  to  participate  in  the  side- 
by-side  opacity  method  com¬ 
parison  during  the  first  day  of 
smoke  school.  Table  5  provides 
a  summary  of  the  DOCS  statisti¬ 
cal  data  generated  from  the  Au¬ 
gusta  smoke  school  field  tests. 

Over  the  full  range  of  opac¬ 
ity  for  black  smoke,  neither  the 
DOCS  nor  the  EPA  Reference 
Method  9-certified  human  ob¬ 
servers  were  able  to  achieve  the 
opacity  error  rate  established  by 
EPA  Reference  Method  9.  The 
mean  opacity  deviation  re¬ 
corded  by  the  DOCS  for  black 
smoke  was  8.6%,  while  certified 
readers  recorded  a  mean  opac¬ 
ity  deviation  of  8.4%. 


Despite  the  inability  of  the  DOCS  to  achieve  the 
accuracy  requirements  specified  for  supporting  method 
equivalency  during  the  Georgia  field  tests,  the  variability 
in  the  DOCS  analyses  was  much  less  than  that  estimated 
for  certified  smoke  readers.  The  95%  confidence  interval 
about  the  mean  deviation  for  the  DOCS  was  estimated  to 
be  8.4 -8.9%  while  that  for  EPA  Reference  Method  9  cer¬ 
tified  human  observers  ranged  from  6.2  to  10.6%.  These 
data  demonstrate  that  the  DOCS  technology  is  more  pre¬ 
cise  and  reproducible  than  human  observers.  However, 
over  the  limited  opacity  range  of  0-60%  for  black  smoke, 
certified  readers  appeared  to  be  more  accurate  than  the 
DOCS  with  a  reported  mean  opacity  deviation  of  6.1% 
compared  with  8.2%  estimated  for  the  DOCS.  The  reasons 
for  the  improved  accuracy  of  certified  smoke  readers  to 
quantify  opacity  under  the  limited  opacity  range  was 
unclear.  The  likely  possibility  is  that  the  Method  9- 
certified  human  observers  utilized  a  background  other 
than  sky  to  improve  the  level  of  color  contrast  exhibited 
by  the  plume  (e.g.,  trees,  telephone  poles,  billboard  signs). 

Although  the  use  of  objects  other  than  sky  in  esti¬ 
mating  plume  opacity  at  a  smoke  school  setting  is  per¬ 
missible,  the  practice  does  not  reflect  the  environment 
in  which  Method  9-certified  readers  would  actually  ap¬ 
ply  the  method  in  practice.  The  smoke  school  stack  in 
an  EPA-approved  smoke  school  field  test  is  —15  ft  above 
ground  level.  At  this  height,  objects  such  as  trees,  tele¬ 
phone  poles,  billboard  signs,  and  so  on  are  normally 
within  the  line  of  site  such  that  they  can  be  employed 
by  a  human  observer  to  improve  visual  contrast,  giving 
the  human  observer  an  enhanced  ability  of  "passing" 
smoke  school.  On  the  other  hand,  under  normal  cir¬ 
cumstances  in  which  a  Method  9  evaluation  in  the 
field  would  be  conducted,  the  visual  observer  typically 


Table  5.  Statistical  data  summary  of  Augusta  smoke  school  DOCS  evaluation. 


Color  of  Smoke- 

Opacity 

Mean  Deviation 

Number  of 

Opacity  Measurement  Approach 

Range 

(%) 

Samples 

95%  Cla 

99%  Clb 

Black— DOCS 
Black — certified  observers 
Black— DOCS 
Black — certified  observers 
Black— DOCS 
Black — certified  observers 
-DOCS 

-certified  observers 
-DOCS 

-certified  observers 
-DOCS 

-certified  observers 


White- 

White- 

White- 

White- 

White- 

White- 


0-100% 

0-100% 

0-60% 

0-60% 

0-40% 

0-40% 

0-100% 

0-100% 

0-60% 

0-60% 

0-40% 

0-40% 


8.6 

8.4 
8.2 
6.1 

7.9 
4.7 

13.2 

6.2 

8.5 

4.9 
7.2 
4.1 


4949 

543 

3620 

398 

2896 

315 

3535 

365 

2565 

265 

2203 

227 


8. 4- 8.9 
6.2-10.6 
7. 9-8. 5 

5. 4- 6. 9 

7.5- 82 

4. 1- 5. 4 
12.6-13.8 

5.4-6, 9 

8. 1- 8. 9 
4. 3-5. 6 
6. 8-7. 6 

3.6- 4. 7 


8.3- 9 

5.5- 11.3 

7. 8- 8. 6 
5.2-7. 1 

7. 4- 8. 3 

3. 8- 5. 6 
12.5-13.9 

52-72 

8-9.1 

4-5.8 

6. 6- 7. 7 

3. 4- 4. 9 


aCI — 95%  confidence  interval;  bCI — 99%  confidence  interval. 
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is  measuring  the  plume  opacity  generated  by  a  stack 
located  on  top  of  a  building  or  a  stack  that  may  be 
hundreds  of  feet  in  height. 

With  respect  to  its  ability  to  measure  the  opacity  of 
white  plumes,  the  DOCS  was  capable  of  achieving  the  EPA 
Reference  Method  9  error  rate  only  when  the  opacity 
range  was  limited  to  between  0  and  40%.  Conversely,  the 
EPA  Reference  Method  9-certified  human  observers  were 
able  to  meet  the  EPA  Reference  Method  9  error  rate  over 
the  full  range  of  opacity  when  analyzing  white  smoke 
plumes.  Again,  although  it  is  not  entirely  clear  why 
Method  9-certified  human  observers  were  able  to  perform 
significantly  better  than  the  DOCS  when  reading  white 
smoke  plumes  during  the  Augusta,  GA,  smoke  school,  it  is 
highly  probable  that  with  the  large  number  of  tall  trees 
and  other  vegetation  surrounding  the  Georgia  smoke 
school  venue,  the  Method  9-certified  human  observers 
were  able  to  strategically  position  themselves  to  view  the 
white  plumes  using  the  dark  foliage,  which  essentially 
provided  a  more  effective  contrasting  background  than 
the  use  of  sky  for  viewing  white  smoke  plumes.  However, 
even  with  less  color  contrast  available  to  evaluate  opacity, 
the  DOCS  was  able  to  achieve  the  Method  9  opacity  error 
rate  over  the  opacity  range  of  regulatory  interest  (i.e., 
0-40%). 

Columbus,  OH,  Smoke  School 

The  EPA-approved  Ohio  smoke  school  was  conducted 
over  3  days  from  March  26  through  March  28,  2002.  The 
climatic  conditions  recorded  during  that  time  period  are 
recorded  in  Table  6.  The  venue  for  the  Columbus,  OH, 
smoke  school  was  a  vehicular  parking  lot  located  in  one  of 
the  city's  municipal  parks.  Weather  conditions  during  the 
Ohio  field  tests  were  considerably  different  from  those 
during  either  the  Utah  or  Georgia  smoke  schools.  The 


Table  6.  Climatic  parameters  measured  or  reported  during  the  DOCS  field  tests — 
Columbus,  March  26 -March  28,  2002. 


Day  of  Test 

Day  1 

Day  2 

Day  3 

Mean  temperature  (°F) 

32 

33 

37 

Average  wind  speed  (mph) 

7.8 

6.9 

5.1 

Max  wind  speed  (mph) 

13.8 

11.5 

13.8 

Wind  direction 

NW 

NW 

S 

Sky  conditions 

Freezing  rain, 

Haze, 

Overcast, 

overcast 

overcast 

scattered  clouds 

Relative  humidity  (%) 

93 

83.2 

95 

Visibility  (mi) 

5.5 

8.5 

5.7 

Barometric  pressure  (in.  of  Hg) 

29.98 

30.18 

30.15 

Precipitation  (in.) 

0.77 

0 

0 

Vertical  sun  angle  (°) 

No  sun 

38 

46 

climate  in  Columbus  for  the  field  tests  was  characterized 
by  freezing  temperatures,  light  rain  mixed  with  snow,  and 
thick,  overcast  skies. 

Quantitative  Analysis.  Like  the  Augusta  smoke  school,  in 
the  analysis  of  the  Columbus  smoke  school  DOCS  data, 
the  DOCS  brush  function  was  not  used  to  evaluate  any  of 
the  digital  photographs.  Furthermore,  because  all  of  the 
DOCS  camera  operators  also  were  certified  as  EPA  Refer¬ 
ence  Method  9  human  observers,  a  larger  number  of  EPA 
Reference  Method  9-certified  human  observers  were  avail¬ 
able  to  participate  in  the  side-by-side  opacity  method 
comparison  conducted  during  the  first  day  of  smoke 
school.  A  summary  of  the  DOCS  statistical  data  obtained 
from  the  Columbus  smoke  school  is  presented  in  Table  7. 

Over  the  full  range  of  opacity  for  both  black  and 
white  smoke,  neither  the  DOCS  nor  the  EPA  Reference 
Method  9-certified  human  observers  were  able  to  achieve 
the  error  rate  established  by  EPA  Reference  Method  9. 
The  mean  opacity  deviation  recorded  by  the  DOCS  for 
black  smoke  was  10.9%,  while  EPA  Reference  Method 
9-certified  readers  recorded  a  mean  opacity  deviation  of 
12%.  Similarly,  for  white  smoke,  the  mean  opacity  devi¬ 
ation  recorded  by  the  DOCS  was  21.6%,  while  EPA  Refer¬ 
ence  Method  9-certified  human  observers  recorded  a 
mean  opacity  deviation  of  10%. 

The  failure  of  both  the  DOCS  and  EPA  Reference 
Method  9-certified  human  observers  to  achieve  the  EPA 
Reference  Method  9  opacity  error  rate  was  unexpected  but 
not  surprising  given  the  challenging  weather  conditions. 
In  other  words,  when  climatic  conditions  are  unfavorable 
for  Method  9-certified  human  observers  to  measure  opac¬ 
ity,  the  DOCS  also  experiences  difficulty.  A  common  con¬ 
cern  expressed  by  many  of  the  smoke  school  participants 
was  their  inability  to  actually  visualize  the  plume  using 
the  sky  as  background.  Technical  guidance  provided  by 
the  EPA-approved  smoke  school  contractors  states  that 
when  faced  with  a  situation  where  there  is  choice  of 
backgrounds,  the  human  observer  should  always  choose 
the  one  that  provides  the  greatest  contrast  because  it  will 
permit  the  most  accurate  opacity  reading.5  Neither  the 
technical  guidance  nor  the  EPA  Reference  Method  9  spec¬ 
ifies  sky  as  the  only  suitable  background  that  can  be  used 
to  quantify  opacity.  However,  by  utilizing  a  smoke  gen¬ 
erator  with  a  15 -ft  stack  height,  establishing  a  more  suit¬ 
able  background  than  sky  is  more  likely  in  a  real-world 
application  of  Method  9  in  which  the  stack  can  be  more 
than  several  hundred  feet  in  height.  Under  these  latter  con¬ 
ditions,  opacity  must  be  read  against  a  sky  background. 

Another  concern  raised  by  smoke  school  participants 
was  their  inability  to  visually  identify  the  position  of  the 
sun  relative  to  the  smoke  stack.  EPA  Reference  Method  9 
requires  that  the  sun  be  at  the  back  of  the  observer  when 
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Table  7.  Statistical  data  summary  of  state  of  Ohio  smoke  school  DOCS  evaluation, 


Color  of  Smoke — 

Opacity 

Mean  Deviation 

Number  of 

Opacity  Measurement  Approach 

Range 

(%) 

Samples 

95%  Cla 

99%  Clb 

Black— DOCS 

Black — certified  observers 

Black— DOCS 

Black — certified  observers 

Black— DOCS 

Black — certified  observers 

White— DOCS 

White — certified  observers 

White— DOCS 

White — certified  observers 

White— DOCS 

White — certified  observers 


0-100% 

0-100% 

0-60% 

0-60% 

0-40% 

0-40% 

0-100% 

0-100% 

0-60% 

0-60% 

0-40% 

0-40% 


10.9 

12 

9.4 
10.6 

8.1 

9.2 

21.6 

10 

15 

9.7 

12.3 

9.5 


3498 

1492 

3066 

1192 

2753 

1012 

4394 

1500 

3758 

1176 

3131 

1020 


10.6- 11.3 
11.4-12.5 

9. 0-9. 7 
10-11.1 

7. 7- 8. 4 

8. 7- 9. 8 
21-22.2 
9.5-10.5 

14.6- 15.4 
9.2-10.3 
12-12.6 

9-10.1 


10.4-11.5 

11.3- 12.7 

8. 9- 9. 8 

9.9- 11.3 
7. 6-8. 5 
8.5-9. 9 

20.8- 22.4 
9.4-10.6 

14.4- 15.5 
9.1-10.4 

11.9- 12.7 
8.8-10.3 


aCI — 95%  confidence  interval;  bCI — 99%  confidence  interval. 

quantifying  plume  opacity.  Although  there  are  various 
methods  to  determine  the  exact  position  of  the  sun  dur¬ 
ing  cloudy  or  overcast  days  given  an  individual's  position 
(i.e.,  latitude,  longitude),  time  of  year,  and  time  of  day,  no 
efforts  beyond  visual  observations  were  made  to  deter¬ 
mine  sun  location  during  DOCS  field  evaluation.10  There¬ 
fore,  field  personnel  could  not  verify  compliance  with  this 
aspect  of  the  method. 

Finally,  although  EPA  Reference  Method  9  does  not 
specify  when  visual  opacity  measurements  should  not  be 
taken,  under  climatic  conditions  that  result  in  a  less  con¬ 
trasting  background,  the  apparent  plume  opacity  can  ap¬ 
proach  0.5  As  a  result,  significant  negative  bias  (error)  can 
occur  when  an  attempt  is  made  to  measure  plume  opacity 
under  severely  cloudy  or  overcast  sky  conditions.  Finally, 
under  dark  overcast  sky  conditions  or  any  climatic  con¬ 
ditions  in  which  visual  contrast  between  the  plume  and 
sky  can  not  be  readily  established,  it  is  more  likely  that  a 
visual  opacity  determination  should  not  be  made  regard¬ 
less  of  the  method  employed. 

Hypothesis  Testing 

Although  a  defensible  evaluation  of  the  equivalency  of 
any  alternative  method  to  EPA  Reference  Method  9  can  be 
achieved  by  comparing  the  mean  opacity  deviation  and 
its  confidence  interval  to  the  established  EPA  Reference 
Method  9  error  rate,  environmental  regulators  have  ex¬ 
pressed  their  preference  for  the  application  of  a  more 
formalized  significance  test  when  addressing  method 
equivalency.611  In  significance  testing,  a  null  hypothesis 
(HJ  is  developed  that  will  be  assumed  to  be  true  in  the 
absence  of  strong  quantitative  evidence  to  the  contrary. 
The  H0  for  the  present  study  may  be  stated  as  follows: 
“The  true  mean  difference  between  the  transmissometer  and 
the  DOCS  opacity  measurements  is  greater  than  7.5%."  This 


statement  reflects  the  assump¬ 
tion  that,  in  the  absence  of 
strong  quantitative  data  to  the 
contrary,  the  two  opacity  mea¬ 
surement  methods  will  be  as¬ 
sumed  to  be  different.  Simi¬ 
larly,  the  alternative  hypothesis 
(Ha)  may  be  constructed  as  fol¬ 
lows:  “The  true  mean  difference 
between  the  transmissometer  and 
the  DOCS  opacity  measurements 
is  equal  to  or  less  than  7.5%.” 
The  rationale  for  constructing 
the  null  and  alternative  hy¬ 
potheses  in  this  fashion  is  to 
shift  the  burden  of  proof  for 
demonstrating  EPA  Reference 
Method  9  equivalency  to  the 
strength  of  the  DOCS  field  data.  In  other  words,  in  the 
absence  of  field  data  that  strongly  support  the  rejection  of 
the  null  hypothesis,  the  conclusion  drawn  from  the  data 
will  be  that  the  DOCS  is  not  equivalent  to  EPA  Reference 
Method  9.  Alternatively,  if  the  strength  of  the  data  were 
sufficient  to  reject  the  H0  (acceptance  of  Ha),  the  conclusion 
drawn  from  this  study  will  be  that  the  DOCS  is  statistically 
equivalent  to  EPA  Reference  Method  9.  The  null  and  alter¬ 
native  hypotheses  may  be  presented  mathematically  as  il¬ 
lustrated  by  eqs  1  and  2,  respectively.  (Note:  80  is  the  tme 
mean  difference  between  the  DOCS  and  actual  plume  opac¬ 
ity  as  measured  by  the  EPA-certified  transmissometer.) 

Null  Hypothesis  (HJ:  80  >  7.5  (1) 

Alternative  Hypothesis  (Ha):  80  <  7.5  (2) 

Given  a  desired  level  of  significance,  a,  and  degrees  of 
freedom  ( n  -  1),  eqs  3  and  4  define  the  critical  t  value  and 
test-statistic  (ftest),  respectively.  The  values  of  these  two 
parameters  are  compared  with  one  another  to  determine 
whether  the  strength  of  the  data  is  sufficient  to  reject  the 
null  hypothesis  (Table  8). 


Critical  t- value:  tcr]tt( 


•crit*'(a/2),n  -  1 


Test  statistic:  he 


3-8 o 

5d 

W 


(3) 


(4) 


where  n  is  number  of  paired  measurements  (i.e.,  DOCS 
and  transmissometer);  d  is  mean  difference  between  the 
DOCS  and  transmissometer  measurements;  sd/\/n  is  stan¬ 
dard  error;  and  sd  is  square  root  of  variance. 
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Table  8.  Test  conditions  used  for  evaluating  the  statistical  equivalency  of  the  DOCS 
to  EPA  Reference  Method  9. 


(a)  Test  Condition:  If  ftest  <  4rit,  then  the  null  hypothesis,  H0,  is  rejected  (accept  Ha  - 
DOCS  is  statistically  equivalent  to  EPA  Reference  Method  9). 

(b)  Test  Condition:  If  ftest  >  fcrit,  the  data  do  not  support  rejection  of  H0  (accept  H0  - 
DOCS  is  not  statistically  equivalent  to  EPA  Reference  Method  9). 


In  practical  terms,  rejection  of  the  null  hypothesis  is 
tantamount  to  accepting  the  DOCS  as  an  equivalent 
method  to  EPA  Reference  Method  9,  while  failure  to  reject 
the  null  hypothesis  essentially  means  that  the  DOCS  is 
not  a  statistically  equivalent  method.  Table  9  summarizes 
the  results  of  hypothesis  testing  using  data  obtained  from 
the  three  EPA-approved  smoke  schools. 

Information  drawn  from  hypothesis  testing  yielded 
mixed  results  with  respect  to  the  equivalency  of  the  DOCS 
to  EPA  Reference  Method  9.  In  all  but  one  opacity  range 
(white  plumes  with  true  opacity  levels  in  excess  of  60%), 
the  data  from  the  Utah  smoke  school  supported  rejection 
of  the  null  hypothesis,  indicating  that  the  conclusion 
drawn  from  that  data  set  should  be  that  the  DOCS  is 
statistically  equivalent  to  EPA  Reference  Method  9.  Con¬ 
versely,  in  the  Georgia  and  Ohio  smoke  schools,  the 
DOCS  field  data,  in  general,  failed  to  support  the  rejection 
of  the  null  hypothesis.  However,  drawing  the  conclusion 


that  the  DOCS  is  not  equivalent  to  EPA  Reference  Method 
9  under  cloudy  or  overcast  sky  conditions  may  not  be 
defensible  given  the  fact  that  the  DOCS  is  limited  to 
utilizing  sky  as  background  and,  under  conditions  en¬ 
countered  at  EPA-approved  Method  9  smoke  schools,  cer¬ 
tified  human  observers  are  not. 

The  ability  of  the  DOCS  to  accurately  estimate  plume 
opacity  is  dependent  on  the  visual  contrast  provided  by 
sky  conditions.  However,  under  the  smoke  school  sce¬ 
nario,  certified  human  observers  are  less  dependent  on 
sky  conditions  because  the  smoke  generators  employed  at 
EPA-approved  smoke  schools  have  relatively  low  stack 
heights  that  afford  human  observers  the  possibility  of 
selecting  any  background  with  which  to  estimate  opacity. 

DISCUSSION 

Data  from  the  DOCS  field  study  indicated  that  digital 
imagining  technologies  have  the  potential  to  accurately 
and  reliably  measure  the  opacity  of  smoke  plumes  when 
weather  conditions  provide  optimum  color  contrast  be¬ 
tween  plume  and  background.  Under  clear  blue  sky  con¬ 
ditions,  the  DOCS  was  able  to  consistently  meet  the  EPA 
Reference  Method  9  performance  standard  and,  over  opac¬ 
ity  ranges  of  regulatory  importance  (i.e.,  0-40%  opacity), 
the  DOCS  accuracy  was  significantly  better  than  that  achieved 
by  EPA  Reference  Method  9-certified  human  observers. 


Table  9.  Summary  of  hypothesis  testing  performed  at  the  0.05  significance  level. 


Smoke 

School 

Color  of 

Smoke 

Opacity 

Range 

(%) 

Mean3 

Deviation 

(%) 

No.  of 

Samples 

M 

Significance 

Level 

(<*) 

Critical11 

T  Value 

Test 

Statistic 

Rejection0 
of  Null 

Hypothesis? 

UT 

Black 

0-100 

6.4 

2336 

0.05 

1.96 

-6.77 

Yes 

Black 

0-60 

5.6 

1957 

0.05 

1.96 

-12.08 

Yes 

Black 

0-40 

5.4 

1745 

0.05 

1.96 

-12.5 

Yes 

White 

0-100 

10 

2405 

0.05 

1.96 

10.39 

No 

White 

0-60 

6.7 

1897 

0.05 

1.96 

-4.35 

Yes 

White 

0-40 

5.9 

1686 

0.05 

1.96 

-8.99 

Yes 

GA 

Black 

0-100 

8.6 

4949 

0.05 

1.96 

8.06 

No 

Black 

0-60 

8.2 

3620 

0.05 

1.96 

4.49 

No 

Black 

0-40 

7.9 

2896 

0.05 

1.96 

2.22 

No 

White 

0-100 

13.2 

3535 

0.05 

1.96 

19.93 

No 

White 

0-60 

8.5 

2565 

0.05 

1.96 

4.72 

No 

White 

0-40 

7.2 

2203 

0.05 

1.96 

-1.47 

Yes 

OH 

Black 

0-100 

10.9 

3498 

0.05 

1.96 

16.83 

No 

Black 

0-60 

9.4 

3066 

0.05 

1.96 

10.24 

No 

Black 

0-40 

8.1 

2753 

0.05 

1.96 

3.53 

No 

White 

0-100 

21.6 

4394 

0.05 

1.96 

46.81 

No 

White 

0-60 

15 

3758 

0.05 

1.96 

36.32 

No 

White 

0-40 

12.3 

3131 

0.05 

1.96 

28.67 

No 

Employment  of  the  brush  function  by  the  DOCS  users  was  not  permitted  during  digital  photograph  analysis;  bFrom  standard  statistical  tables,  for  a  =  0.05  and  n>  1 20,  t critical  is 
^1 .96;  cWhere  the  null  hypothesis  is  rejected,  the  data  indicate  that  DOCS  is  equivalent  to  EPA  Reference  Method  9.  Conversely,  if  the  null  hypothesis  Is  not  rejected,  the  conclusion 
is  that  DOCS  is  not  equivalent  to  EPA  Reference  Method  9. 
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The  decision  not  to  employ  the  DOCS  brush  function 
during  the  field  trials  may  have  limited  unfairly  the  digital 
technology  from  demonstrating  its  full  performance  ca¬ 
pability.  Conceivably,  a  DOCS  technology  user  employ¬ 
ing  the  brush  function  would  have  had  the  opportunity 
to  cut  and  paste  a  darker  background  with  which  to  eval¬ 
uate  white  smoke  plumes  and  a  lighter  background  to 
evaluate  the  opacity  of  black  smoke  plumes.  By  improv¬ 
ing  the  contrast  between  the  plume  emissions  and  back¬ 
ground,  a  more  accurate  opacity  reading  can  be  made. 
Moreover,  using  the  digital  technology  allows  an  inter¬ 
ested  party  the  ability  to  reevaluate  how  the  plume  back¬ 
ground  was  established  after  the  initial  opacity  measure¬ 
ment  has  been  reported.  It  is  anticipated  that  once  the 
technical  implications  of  utilizing  the  brush  function 
have  been  fully  explored  and  documented,  a  future  opac¬ 
ity  assessment  of  the  smoke  school  digital  photographs 
using  the  brush  function  may  be  warranted. 

CONCLUSIONS 

The  results  for  the  three  smoke  school  field  tests  suggest 
that  climatic  conditions  have  a  profound  effect  on  the 
ability  of  the  DOCS  and  Method  9-certified  smoke  readers 
to  meet  the  performance  criteria  established  by  EPA  Ref¬ 
erence  Method  9.  During  the  Utah  field  test,  sky  condi¬ 
tions  were  clear,  which  provided  optimal  color  contrast 
between  plume  and  sky.  Under  these  conditions,  the 
DOCS  error  rate  was  significantly  less  than  that  estab¬ 
lished  under  EPA  Reference  Method  9.  Moreover,  the 
DOCS  opacity  readings  were  consistently  more  accurate 
than  those  recorded  by  Method  9-certified  human  observ¬ 
ers.  Alternatively,  when  smoke  plumes  were  viewed  under 
weather  conditions  characterized  by  cloudy  or  overcast 
skies,  the  DOCS  and  Method  9-certified  human  observers 
had  difficulty  in  accurately  quantifying  opacity.  This  re¬ 
duced  ability  to  measure  opacity  was  not  surprising  given 
the  fact  that,  as  color  contrast  between  the  plume  and  sky 
conditions  diminishes,  apparent  opacity  also  decreases. 
When  the  DOCS  and  human  observers  are  compelled  to 
use  sky  as  background,  the  accuracy  of  the  DOCS  in  quan¬ 
tifying  plume  opacity  clearly  surpasses  that  of  Method 
9-certified  observers.  However,  under  conditions  encoun¬ 
tered  at  EPA-approved  smoke  schools,  human  observers 
can  often  take  advantage  of  the  relatively  low  stack  height 
of  smoke  generators  to  employ  objects  other  than  sky 
(e.g.,  trees,  light  poles)  with  which  to  quantify  opacity. 
This  ability  represents  a  clear  advantage  for  human  ob¬ 
servers  in  their  attempt  to  pass  smoke  school  but  does  not 
reflect  the  typical  Method  9  field  application  under  which 
plume  opacity  from  stacks  with  heights  much  greater 
than  the  local  tree  line  can  be  estimated.  Under  these 
conditions,  the  sky  must  be  used  as  a  visual  background. 
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